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[57] ABSTRACT 

A layer based paint system permits painting of an image 
raster of a composite image while viewing subsequent 
processing on the image raster. The composite image is 
mapped to an image processing tree. Each of the elements io 
the image processing tree are defined by an associated data 
structure that identifies their parent and children elements in 
the tree. Each of the elements may be, for example, an image 
element or a process element. Associated with each image 
element is an image raster and a transform. Viewing and 
painting of an image element may be done at any level in the 
image processing tree. Painting of the element directly 
modifies the pixels in the image raster. The pixel coordinates 
of brush strokes in the viewed image are transformed into 
pixel coordinates of the image raster material by applying 
) the inverse transform of the viewed layer, the image element 
layer and all of the intermediate layers to the pixel coordi- 
nates of the brush strokes. The profile of the brush stamp is 
also transformed by applying the inverse transforms of the 
image layer and all intermediate layers to the pixel coordi- 
nates of the brush stamp. Because the process elements are 
separated from the image elements in the image processing 
tree, painting may be performed on any layer while viewing 
the effects of subsequent processing. 
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IMAGE COMPOSITION SYSTEM AND remove the rotation, the process of blurring must first be 

PROCESS USING LAYERS undone before the rotation is reversed. Once the rotation is 

reversed, the processing is reapplied to the image. The 

FIELD OF THE INVENTION iterative reprocessing steps performed when modifying an 

5 image layer are undesirably time consuming. In addition, 

This invention relates generally to image editing systems repe titive reprocessing of the image may result in the 

and more particularly to a layer based paint system . • addition of visual artifacts that detract from the quality of the 

BACKGROUND 6n*m»&. , 

More advanced layered paint and animation systems use 

The use of computers to change, augment, and reshape 10 object based architectures. Each layer is an object that may 

original film and video images has been a practice that has contain an image, an image process (e.g., a blur) that applies 

steadily been on the rise. A digital film workstation (DFW) a function to the image, or brush strokes, which change the 

may be used to fix problems that originate during the color of selected pixels within the image. Each object retains 

production stage and to create special effects. Many special information about its position, scale, rotation, etc. Because 

effects sequences are achieved by layering different images 15 image layers are objects that contain an original image, 

together in one composite frame. A first film or video clip rotation and scaling information, there is no longer a simple 

may show an actor in a first location and a background relationship between the gestures a user might make with the 

image of a second location may be layered under the actor. input device and the corresponding changes that would have - 

The resulting affect is that the actor appears in the second to be made to paint directly on the image raster. As a result, 

location. A composite image may be a combination of tens 20 object based paint systems typically dictate that any new 

of different image layers depending upon the desired special paint is applied to a new layer. Accordingly, in such systems 

effect. it is not generally possible to directly modify the image 

Frequently it is desirable to add or remove a portion of an raster associated with the object without creating a new, 

image from one of the layers of the composite image. For modified object. Rather, any modifications to the image 

example it is often necessary to remove unwanted elements, 25 raster are made in layers above the image raster, 

such as the safety wires securing the stunt people, from film Known object based systems do not typically allow the 

sequences. Adding or removing elements from an image effects of processes to be viewed when adding brush stroke 

requires that a film editor be able to manipulate individual layers beneath the process layers. Thus, if a user wishes to 

pixels within the composite image. One software tool that modify an image layer, the painted layer is typically gener- 

enables a film editor to manipulate individual pixels in the 30 ated above the process layers, and when the painting is 

composite image is a layer based paint system. complete the layers are reordered and the processes may be 

Typical layer based paint systems include an input device, viewed. Because processing is not done directly to the 

such as a mouse or pressure sensitive tablet coupled to a images, there is no degradation of the image. However, the 

pointer on a display screen. The movement of the input 35 additional layering steps are time consuming and inconve- 

device mirrors the movement of the pointer so that the user nient. 

is given the impression of painting directly on an image on TNVFNTION 

the display screen. The layer based paint system allows users SUMMARY OF THE INVEN HON 

to paint new layers, creating new images with their own A layer Dase ^ pa j n t system permits painting of an image 

image and transparency (alpha) information. The layer ^ raster of a composite image while viewing subsequent 

based paint system also allows the user to paint on existing processing on the image raster. The composite image is 

layers, changing the image and/or alpha information in the mapped to an image processing tree. Each of the elements in 

layer to add or remove objects from the layers. Using the me iinage p roceS sing tree are defined by an associated data 

layer based paint system, it is also often possible to move the structure that identifies their parent and children elements in 

layers relative to each other to give them an offset in X and 4S ^ (rce Each of me dements may be, for example, an image 

Y directions. element or a process element. Associated with each image 

In the simple layer paint systems, each layer is itself an element is an image raster and a transform. Viewing and 

image. The layers are stored as image rasters with image and painting of an image element may be done at any level in the 

transparency (alpha) information. If a process is performed image processing tree. Painting of the element directly 

on the layer, the process directly modifies the pixels asso- 50 modifies the pixels in the image raster material. The pixel 

ciated with the layer. When the process has been performed coordinates of brush strokes in the viewed image are trans- 

on that layer, the layer is indelibly modified. formed into pixel coordinates of the image raster material by 

One advantage of the simple layer based system is that, applying the inverse transform of the viewed layer, the 

since each layer is simply an image raster, it is possible to image element layer and all of the intermediate layers. The 

paint directly onto any layer to change the contents of rasters 55 profile of the brush used for the brush stroke is also 

in any layer. There is always a simple one to one relationship transformed by applying the inverse transforms of the image 

between the gestures made using the input device and the layer and all intermediate layers. Because the process ele- 

changes to be made upon the image. The one to one ments are separated from the image elements in the image 

relationship makes it possible to paint directly onto any layer processing tree, painting may be performed on any layer 

even if there are layers above it and below it, 60 while viewing the effects .of subsequent processing. 

One problem with the layer based paint system is that if In one aspect, a computer readable medium for storing 

the artist subsequently wishes to modify a layer, first the data accessible by an application program during execution 

processing of the layer must be undone. For example, if the of the application program on a data processing system is 

artist rotates a layer, the rotation is applied to the pixels in provided. The computer readable medium includes an image 

the image raster to provide an image layer with new, rotated 65 processing data structure stored in the memory and repre- 

pixel values. To blur the image, a blurring process is applied, senting a composite image, the image processing data struc- 

updating the image raster with a new, blurred image. To ture comprising a hierarchically linked coupling of ele- 
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ments. Each of the elements include a type indicator In another aspect, a graphical user interface for painting 
identifying a type of the associated element, a reference to a composite image is provided. The composite image 
a parent element in the image processing data structure, and includes one or more hierarchically layered elements corn- 
references to any child element in the image processing data prising image elements and process elements. The graphical 
structure. The types of elements include process elements 5 user interface includes a display window for displaying an 
and image elements, and an order of the hierarchically image, a hierarchy window for displaying an image pro- 
linked coupling is related to an order of layering of the cessing tree representing the hierarchically layered elements 
elements in the composite image. forming a composite image, means for selecting a layer in 
In another aspect, a method of displaying a composite ^ hierarchical layer to be displayed in the display window, 
image represented by a hierarchically ordered coupling of w means for modifying a first image element at a first layer 
elements is provided. The elements include a pointer to a q£ ^ com ^ site image whilc displaying ^cond image 
parent element and at least one pointer to a child element. dement ^ a sccQQd nh ^ ci ^ } of ^ osite 
The elements are apportioned into classes including image 
elements and process elements with the image elements im a S e - 

including an image source and a transform. The method BRIEF DESCRIPTION OF THE DRAWING 

includes the steps of: a) designating all the elements at the 

lowest level of the hierarchically ordered coupling of ele- In the drawings, 

ments as current working elements, b) for each current FIG. 1 illustrates a number of layers forming a composite 

working element that is an image element, transform the image; 

image source using the associated image element transform, - A .„ . . . , 4 . . 

x f , B , , . i .... • on FIG. 2A illustrates an image processing data structure; 

c) for each current working element that is a process 20 ° r ° 

element, process the coupled children according to a process FIG- 2B illustrates an example of a composite image 

associated with the process element, d) designate the parent corresponding to the image processing data structure of FIG. 

elements of the working elements as the working elements. 2A; 

Repeat steps b through d until all of the elements in the FIGS. 3A-3C illustrate one embodiment a graphical user 

image processing tree have been processed. Display an 25 interface that may be used to build an image processing data 

image defined by the image processing tree on the display structure such as that shown in FIG. 2 A; 

screen - FIG. 4 is a block diagram of an example data structure 

In another aspect, a method of painting pixels of a used in the image processing data structure of FIG. 2A; 

material in a composite image displayed on a display device F[G 5 ^ a Mock dia g ram of one embodiment of a process 

is provided. The painted pixels are selected in response to 30 for building an ^ processirjg data structure using ele- 

pixel coordinate data received from an input device. The men(s defined b {h& ^ stmcture of nG 4; 

composite image is formed from a plurality of hierarchically rT „ ✓ c . t c 

, j 1 * , ♦ ■ 1 j- • 1 ♦ FIG. 6 is a block diagram of one embodiment of a process 

ordered elements, the elements including image elements c L £ . . Tr « 1 j- . u • *• 

. , ' n u ■ 1 * a * a for transformmg brush stroke coordinates when painting a 

and process elements. Each image element identifies a . ... . & . , . , . -f A 

j 'c a u. . c Tn. *u a • ^ a *u l avcr within the image processing data structure of FIG. 2A; 

source modified by a transform. The method includes the 35 7 .««,?. , 

following step of determining, responsive to the pixel coor- r FIG - a bloc * diagram of one embodiment of a process 

dinate data received from the input device, coordinates of for transforming brush profiles for painting a layer within 

pixels in the material to be painted. The step of determining the ima 8 e processing tree of FIG. 2A; and 

further includes the step of storing the pixel coordinate data FIG. 8 is a diagram of a brush profile that may be 

received from the input device as working pixel coordinate 40 transformed using the process described in FIG. 7. 

data. For each image element between the image element in DETAILED DESCRIPTION 
the hierarchy corresponding to the displayed composite 

image and the image element associated with the material to A paint application is provided for editing and animating 

be painted, inclusive, the method includes the steps of image data. A frame of film or video is formed from a 

a)extracting the transform associated with the image 45 composite of a number of different image layers. The image 

element, b) generating the inverse transform of the extracted layers may be bit maps provided with the software paint 

transform; and c) applying the inverse transform to the application, still images retrieved from an analog frame of 

working pixel coordinate data, and d) painting the source of video, matte backgrounds painted by a user, etc. Each of the 

the image element by modifying the values the pixels of the image layers is formed from a matrix of pixels. Each pixel 

source using the working pixel coordinate data. 50 in the matrix includes, among other values, color informa- 

In another aspect, a method of adjusting a brush profile for tion indicating a color of the pixel and alpha information, 

painting pixels of an image raster in a composite image Alpha information identifies the transparency of the pixel 

displayed on a display device is provided. The composite and thus indicates which portions of the image are viewable 

image is formed from a plurality of hierarchically ordered when they are layered in the composite image. Thus, when 

elements with the elements including image elements and 55 the image layers are layered, the transparency information 

process elements and with each image element identifying a indicates which portions of the image layers may be viewed 

source modified by a transform. The method includes the by the user on the display device. 

steps of a) generating a brush matrix representing a brush For example, referring now to FIG. 1, an example com- 

profile selected for painting, b) for each image element posite image is shown. The composite image 22 includes a 

between the element of the hierarchy of elements corre- 60 background layer 30, a number of intermediate layers 2 6 and 

sponding to the displayed composite image and the image 28 layered on the background layer 30, and a foreground 

element associated with the image raster to be painted, layer 24 layered on layers 26, 28 and 30. Ihus, the image 

including the image element associated with that image layers form a display hierarchy, where displaying of image 

raster, extracting the transform associated with the image layers that are relatively higher in the hierarchy (such as 

element and applying the extracted transform to the brush 65 foreground layer 24) includes displaying of layers that are 

matrix, and c) inverting the transformed brush matrix to relatively lower in the hierarchy (such as background layer 

provide a brush profile for painting the image raster. 30). 
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A variety of process es a re capable of being performed on 
ea' cn oL mtflffy'efi ' ufTEe composite and/or on groups of th e 
sequential layers, in th^ rnmpnsite imag e~"The types of 
p rocesses include sp atial transform s, filters and paste jpro 
cesses. Spatial transforms include runctionsfor shifting the 5 
i mage rasters in tne'x-y direction, rotating the image rast ers 
and scaling the size of the im ape rasters. Filte j^ processe s 
include colo r trans f orm functions such as tirmng an d blur- 
rig g tne in^ g e rasters. Paste processes (also known as blen d 
pfocessesjmcTude functions for me rging two image r asters 
together or for modifying alpna mtormation of an image 
raster. Other types of processes, known to those in the art, 
such as gamma adjusting, color correctio n etc. also may b e 
used to process the i mage rasters. 

Trie composite image may be displayed at any level in the 15 
hierarchy. For example, a user may display the image 
starting at the hierarchical level including image layer 28. 
With such a selection, the displayed composite would 
include the combination of image raster 28 with image raster 
30. When the composite is displayed from the perspective of ^ 
a selected layer, different layers of each of the image rasters 
that are relatively lower in the hierarchy are viewable 
according to the alpha information in each of the respective 
lower layers. 

Hie composite image also may be accessed for painting at 25 
any level of the hierarchy During painting, the composite 
may be displayed at the same level of the hierarchy at which 
painting occurs or alternatively at a higher level within the 
hierarchy Viewing the composite at a relatively higher layer 
within the hierarchy allows the effects of processes that are 30 
performed at relatively higher hierarchical levels than the 
level at which painting is being performed to be displayed 
during painting. For example, if a tint process is performed 
at layer 24, and a user selected layer 26 for painting, each 
brush stroke that is painted on layer 26 would be displayed 35 
using the tint process associated with layer 24. 

The paint system represents a composite image using an 
image process data structure. In one embodiment, a tree type 
data structure is used. However, the present invention is not 
limited to a tree type data structure and other data structures 40 
that allow elements to be linked in a hierarchical manner, 
such as graph type data structures, may alternatively be 
used. When using an image processing tree data structure, 
each 'leaf' of the tree is associated with an image raster. The 
image raster may be, for example, a stored bit map, a shape 45 
generated by the user using the paint system, text, a digitized 
frame of video or other similar image representation. As 
each of the image rasters is added to the composite and 
processed by the user, the image rasters and processing steps 
are linked together in the image process tree. Thus, the 50 
image process data structure defines the composite image. 

Referring now to FIGS. 2 A and 2B, process tree 50 of 
FIG. 2A is a definition of the composite image 80 in FIG. 
2B. As the user adds layers to the image, the links between 
the image rasters and processes are created in the tree. For 55 
example, the background may be the first layer processed. A 
background element is formed from the background raster 
62 combined with a 2D spatial transform 64. In the com- 
posite of FIG. 2 B, the background 62 is a mountain scene 
and 2D spatial transform 64 rotates the mountain scene by 60 
ninety degrees. The next layer added in the composite 
includes a text raster 52. The text raster is transformed by a 
2D spatial transform 54 (which scales the size of the text) 
and filtered by tint process 56. Paste process 58 is then 
applied to the tinted text raster 526 to update alpha infor- 65 
mation in the tinted text. The paste process 58 acts to remove 
the tint from all but the text portion of raster 526. The 



6 

resulting raster 52c is transformed using spatial transform 
process 60 to move the text in the y direction. Paste process 
66 combines the resulting image elements 62a and 52a* to 
provide raster 67. Raster 67 therefore reflects the processing 
of image rasters in the first and second layer of the com- 
posite image. 

A shape raster 68 may be generated by the paint tool in 
response to user input. The shape raster 68 is combined with 
raster 67 using paint process 70 to provide raster 71. Raster 

71 thus reflects the processing of image rasters in the first, 
second and third layer of the composite image. A stored bit 
map raster 72 of a bear is next added to the composite. 
Before adding the bear to the composite, the bit map raster 

72 is scaled by 2D spatial transform process 74. The scaled 
raster 12a is then pasted to raster 71 to provide composite 
image 72. The composite image 72 is then processed using 
2D spatial transform 78 to shift the location of the overall 
image within the frame. The result is composite image 80. 

FIGS. 3A-3C illustrate an exemplary graphical user inter- 
faces 81 of a paint application. All of the figures illustrate the 
same frame of a video sequence, where the frame of the 
video sequence is a composite of a number of different 
layers of image elements. However, each figure displays the 
frame from a different viewpoint: FIG. 3 A displays the entire 
composite image of a blurred corkscrew on a mountain 
background, FIG. 3B displays the transformed corkscrew 
element and 3C displays the corkscrew material. 

The graphical user interface of the paint application 
includes a display window 82, a hierarchy window 90 and 
a brush window 100. The display window displays the 
composite image of a frame from a selected viewpoint. The 
display window includes Start time window 84, End time 
window 86 and Duration window 88. The Start time window 
84 identifies the start time of the frame within the video 
sequence. The End time window 86 identifies the end time 
of the frame within the video sequence, while the Duration 
window indicates the number of frames in the video 
sequence. 

The brush window 100 is used to select a brush size, shape 
and color for painting on the image in the displayed window 
80. In particular, the profile control 101 controls the brush 
stamp of the brush as it is stroked over the image, where the 
brush stamp is the number and arrangement of pixels 
affected by each distinct contact of the brush. The input 
control 102 controls the color of the brush. The opacity 
control 103 allows the user to select the transparency of the 
brush, i.e., how much the pixels in the image will be affected 
during painting. The spacing control 104 allows the user to 
select a spacing between repeated stamps along each brush 
stroke. When a brush is selected for painting, the character- 
istics of the brush and associated brush strokes are defined 
relative to the displayed image. Thus, if painting is per- 
formed on one layer, adjustments to the characteristics of the 
brush strokes are made when viewing other layers of the 
composite image. The methods used to adjust the brush 
stroke are discussed below. 

The hierarchy window 90 includes an image processing 
tree 95 that is built by the user of he paint software as the 
composite image is created. The image processing tree 
includes image elements and processes. Image elements 
include digitized film clips or pre -generated bit-maps which 
are represented by film clip icons such as icon 96. Image 
elements also include shapes selected from a shape menu 
accessible from button bar icon 211. Image elements also 
includes composite groups of image elements. For example, 
folder 89 is an image element. 
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The processes include filter-type processes (including selection of one of the rendering controls causes that ren- 

color transform processes such as tint and blur processes) dering operation to be performed on a selected portion of the 

and paste type processes. The filter-type processes are displayed image. Button 225 is a graph button. The selection 

represented by the triangle icon 98, while the paste type D f the 20 graph button 225 results in the display of 

processes are implied by the addition of an image element in 5 animation, time lines and graph editors. Button 226 is a 

the processing tree. brush icon. Selection of the brush icon is used to display 

A variety of spatial transform type processes are associ- brush controls for controlling the size, transparency and 

ated with image elements in the image processing tree. As otncr characteristics of the brush selected from the tool 

mentioned above, the spatial transform type processes chcst Button 227 is the hand icon. The selection of button 

include processes that rotate, scale, move or corner-pin the 10 2 27 causes the properties of a selected element to be 

associated image element. The spatial transform processes displayed. 

are not linked as individual elements in the image processing ^ aboye window and icons are merel example ic0QS 

tree but rather are always implied by their appearance Qf ^ embodiment and not a Umiting factor of ^ present 

adjacent to the image element which they affect. invention. The image processing tree is built by the user 

The graphical user interface 81 also includes a button bar 15 usiQg the aboye described but ton bar controls. 

205. The selection of a button on button bar 205 causes an _ . . . , . , c 

- - c „ The generated composite image may be viewed trom a 

associated software application to perform the following ■ A e . • ¥ i j *T • *i • 

- -» ^ nn +u *u i 4- t u- u variety of perspectives. Included in the image processing 

functions. Buttons 210-214 are icons, the selection of which n l ■ M , fv - ifj 

„ , - c „ . . T tree 95 is an eye icon 91 and a set of View pull down menus 

are generally known to perform the following functions. In _^ ' * t *u u- u- i i i < 

5 7 . A . « i , , cy a t 92 and 94. The eye icon controls the hierarchical level at 

response to selection of button 210, a file is opened. In 2 q ,. . iL . j T cm ia *u 

* ... t 4 . f , )f A zu which the composite unage is viewed. In FIG. 3A, the eye 

response to the selection of button 21 1, a displayed edit *\ . 6 , . , , . , A ' . ' 

j . . , ™ * t . r i_ ' w ' M * * icon 91 is at the highest level m the hierarchy. Accordingly, 

product is saved. The selection of button 212 causes a . to . . . , c ' c ° ' 

r t . . - * • j> i j * i_ j the composite image is being viewed trom the foreground 

selected portion of an image that is displayed to be removed F u n * .u i * .u u v j i« 

c • i . ! , . 6 , • iU • fc t« ^i^ trt „ „f layer down through all of the layers to the background. In 

from the display and stored in the buffer. The selection of / ir?\u . J 01 « ^ , „ , . 

... r ; i * j r ■„ * u FIGS. 3B and 3C, the eye icon 91 has been moved to the 

button 213 causes a selected portion or an image to be 25 ^,3^^^ [ eve i 

copied to the buffer although it remains in the display. The ' 

selection of button 214 extracts a previously saved image or I" addition to the eye icon 91, the View pull-down menus 

image portion from the buffer for pasting into the displayed «? and 94 control what portion of the selected layer is 

. ° r displayed. The view options provided by View pulldown 

Buttons 215-227 are used to provide the following paint 30 me » u « *"*«te Selection and Composite. Accordingly 

functions. Button 215 is an import image icon. Selection of ^ , sel f K cted la y er ., ma y be ™ wed * .f J auon . ° r 

„ . c • ♦ i_ alternatively the composite may be viewed (taking into 

button 215 causes an image or sequence of images to be ^i^y^y vu F y \ & 

, - . - jf. , „ J, ,u ,v 0 r.^ account all other layers below the selected layer). The view 

retneved from a disk for display. As the image is inserted . ' ' . 7 . D1 

. . j. , /.u ■ ■ • tn options provided by View pulldown menu 94 include Ele- 

lnto the display, a name of the image is inserted to the *? r / r t 

uiiu ui ^ & c , ^ - . ^ e ment and Matenal. The Element view is the combination of 

hierarchy. Button 216 is a group icon. Selection ot button 35 . . . ,. A . , tt n ^- ntaA ^ nt: ^ tl . arie 

7 . j. j r 1 * * *i_ u- u ♦ the image raster modified by any associated spatial trans- 

l U ^ A ^^^^f^^^S^ form-TrfeMaterialviewistheimagerasterunaffectedbythe 

be grouped together. For example, the group may be selected example, FIG. 3B illustrates the 

to cause a corkscrew image raster modified by a rotation and scale 

rather than on individual elements on the hierarchy. Button & / : m ™ 

. ™ , ru ** no* a *~ spatial transform. FIG. 3C, which illustrates only the image 

218 is an output icon. The selection of button 218 is used to 40 F , • , . *u 1 • * u c £ 

add an element at the top of the hierarchy, thus defining the raster material, shows the corkscrew image raster before the 

output element of the image processing tree. Button 219 is s P aUal transfonn has been a PP hed ' 

a new image icon. The selection of button 219 causes a new A user may use the eye icon and View pull-down menus 
image element to be added into the hierarchy. Button 220 is to select a desired perspective of the composite image for 
a shape icon. The selection of button 220 causes a fist of 45 dis P la V- Painting may be performed on any element that is 
available shapes to be provided. A user may then select one selected for display. The pixel data in the underlying image 
of the available shapes to paste into the image display. As the raster is dynamically updated during the painting process, 
shape is pasted into the display, the shape element is As a result, when the brush strokes are completed during 
positioned at the associated position in the hierarchy. Button painting of any displayed element, the appropriate pixels are 
221 is an effect icon. The selection of button 221 causes a 50 modified in the underlying image raster such that the pro- 
pull down menu to be viewed which lists the available cessing of the underlying image raster, as defined by the 
effects and processes that may be performed on images or image processing tree, results in the correct image being 
groups of images in the hierarchy. By selecting one of the displayed. 

processes in the pull down menu, that process is applied to For example, in FIG. 3A the corkscrew image raster is 
the displayed image. In addition, the process is inserted to 55 selected for painting, while the selected view is the entire 
the corresponding location in the image processing tree. composite image (as indicated by eye icon 95 and View 
Button 222 is a hammer/brush icon. The selection of pull-down menu 92). An ellipse 83 is painted on the cork- 
button 222 causes a tool chest to be provided for painting screw image raster. Note that the effects of processing 
and shape tools. The painting and shape tools, for example, functions that are relatively higher in the image processing 
include different types of brushes that may be used for the 60 tree hierarchy than the selected element that is painted may 
painting process. Selection of one of these brushes from the be viewed during painting. Thus, the painted ellipse is 
tool chest causes that brush to be used during painting. blurred as a result of Box Blur process 98. 
Button 223 is a hierarchy icon. The selection of button 223 To view the element without viewing the affects of 
causes the hierarchy window to be displayed. Button 224 is processes relatively higher in the hierarchy, the View pull- 
an explanation mark icon. The selection of button 224 65 down menu may be changed from Composite to Selection, 
causes a fist of rendering controls to be displayed. One Thus, in FIG. 3B, the element that includes the corkscrew 
example rendering control, for example, is 3-D rendering. A image raster with the painted ellipse 83 is shown, without 
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the blurring and paste processes that are relatively higher in formats, such as Red, Green, Blue, Alpha (RGBA) format, 
the hierarchy than the corkscrew element. CCIR 601 4:2:2 format or other well known pixel formats. 
The corkscrew element of FIG. 3B still reflects the The MaterialElement class includes ImageElement and Mat- 
processing of the image raster with the rotate and scale tenement sub-classes. The ImageElement sub-class further 
spatial transform applied to the image raster. The image 5 includes a ComposiUonElement sub-class. The. data struc- 
r . ... . t , ... 7 «■ j u c ture of each MaterialElement includes a reference to a 
raster, after it has been painted with the ellipse and before it . , . 4 , * t • , £ . tU *• i 
. . , • ^ j *. • u • <*r> source image and pointer to a 3x3 matrix denning the spatial 

? aS J?* d ^, y r^ DgP ,? 1 ! 0n St T m a-^Z transform Process that is perfomicd on the soiree image. 
In FIG. 3C, the View pull-down menu 94 has been modified ^ sour(x ^ ig ^ an . ^ a sh or 

to select Material rather than Hement. When comparing anolhef ^ elemen , (when the { element js a 

FIG. 3A to FIG. 3C it can be seen that the pixel locations 10 c itionElement) . ^ dita structure of the i mageE le- 

affected by the brush strokes on the composite image are mentsub .class further includes a reference to a paste process 

transformed such that the appropriate p*el locations of the ^ a reference tQ & node ^ ^ ^ essi ^ where 

miage raster are modified during the paint process. The ^ ^ fa connected . ^ MatteElem ent sub-class 

selection of pixel locations for modification in the image identifies Mhi ^ vAi± ide teanspareilC y Mol . 

raster is performed by applying the reverse effects of all of « ^ data rf a c sitionElement Ue[ 

the spatial transforms in the image processing tree between & refelence to me te ^ md isiised to reflect 

the displayed layer (at which painting occurs) and the ^ ^ of elements ^ m nhavA lower in the 

selected element (which includes the image raster) to the hierarchy 

pixel locations indicated by the brush strokes. ™ T T . A r . 4 . , 1 t , t . , , . 

r ^ . „ n The ImageLmkElemeDt is an element that includes a link 

In order to reverse the effects of the spatial transforms to an ImageElement similarly, the MatteLinkElement is an 

between the displayed layer and the selected element, the elcment for providing a ^ t0 any Element. 

inverse spatial transforms are applied to the values of the . . n A , . nrrac nr „ r ^ occ .^ ; ntn *u 0 im ™ 

, . . «• . i . i i . , » ,. As the user adds images or processes into the image 

pixel locations affected by the brush strokes An allowance ^ tree ^ a routine called MakeConnections 

.s ako made for the changing geometry of the brush stroke £ ^ MakcC ; nnection routine forms me appro . 

profile. For example, for a given brush profile, if the image ^ . 4 r , , . , . , , tU . 

r t . t , , . . , c . .7 pnate links between the newly added element and the image 

raster had been scaled down by a factor of two at the rocessi bec b datin the t and chi]dren data ^ 

displayed, parted layer, then the brush profile is scaled up ^ of ^ tQ * c& J ^ addition rf ^ new 

by a factor of two when the stroke is applied to the unage . . . . . . 

ra ^ er element into the image processing tree. 

, , , _ „ A „^ .„ .30 In one embodiment, all of the leaf elements are stored in 

Although the example of FIGS. 3A-3C illustrates only a file ^ kaf elemeQts m ^ tQ idendf starti ints 

one layer affected by one spatial transform a composite fof ocessi branches of the i processmg tree and are 

image may be formed from tens of layers. Each of the layers faU associated ^ an ^ raster? where the ima 

may be processed by an associated spatial transforms, and iQclude ^ di bU and sfa mQ $ 

groups of layers may also be processed by spatial trans- 35 Qne ^ melhod for generating me ^ 

forms. Therefore, the reverse translation process may si tree> At st m one of the kaf elemen(s is 

involve the apphcation of numerous inverse spatial trans- se i ecte d as a CurrentElement for processing. The processing 

forms to the pixel coordinates .of the brush strokes in order 130 _ 142 afe foUowed for each leaf ^ me ^ 

to ensure that the image raster is accurately painted. The data si ^ for u ^ of clarity opl one 

structures mcorporated m the image processing tree allow ^ be described herein At ste 134> the class of the 

for fast accurate update of image raster in response to the 0^^^ is checked to determine whether it is an 

painted brush strokes. ImageElement. If so, at step 138, the source of the image 

One example of a data structure 118 that may be used to dement, which is either image raster material another image 

define an element in the image processing tree is illustrated element (if the ImageElement is a CompositionElement), is 

in FIG. 4. The image processing tree is a tree of coupled 4$ trans f orrrje d according to the spatial transform of the Ima- 

elements. The data structure 118 of each element includes a ge Element. If the CurrentElement is not an ImageElement, 

class/subclass field 120, a reference 122 to a parent element men at step 132 i t ^ determined whether the CurrentElement 

in the tree and a reference 124 to an array of children ^ a p roC essElement. If the CurrentElement is a Process 

elements in the tree. If there are no children associated with Element, then at step 136 the process associated with the 

this element, the value at the location in the array is Null. 5Q CurrentElement is applied to the children associated with the 

The data structure 118 may also include a number of other CurrentElement. If the CurrentElement is not a ProcessEle- 

fields depending upon the class of the element. Available men t or an ImageElement, the process continues at step 135 

classes of elements include: ShapeElements, to determine the type of element and connect the image 

ProcessElements, and MaterialElements. ShapeElements processing tree appropriately. Once the type of the Curren- 

identify one of a set of predefined shape objects that are ss tElement has been determined and the processing of the 

made available with the paint application. A data structure of CurrentElement has occurred, at step 140 it is determined 

a ShapeElement class includes a reference to one of the whether the CurrentElement has an associated Parent Ele- 

predefined shapes. men t. If so, at step 142, the CurrentElement is updated to 

ProcessElements identify a process that is performed on point to the ParentElement, and the processing returns to 

elements that are relatively lower in the hierarchy than the 60 step 134. The process of traversing up the image processing 

process element (i.e, any children of the process element). tree from the leaf to the highest element (i.e., the root 

The functions identified by the process elements include element) continues until all elements in the tree have been 

blurring, tinting, and other filter processes. The data struc- processed. 

ture of a Process Element includes a reference to the The image processing tree that is generated using the 

function associated with the process. 65 MakeConnections routine is used both for display purposes 

MaterialElements identify raster images. Pixel informa- and for processing pixels during paint operations. During 

lion in the raster image may be provided in a variety of painting, a user paints a desired pattern on the viewed image 
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raster. However, because the viewed image raster may have to the image raster material, the profile of the brush that is 

undergone numerous transforms and processing before used to apply the brush strokes is also adjusted to ensure that 

viewing, there is not generally a one to one correspondence the appropriate pixels in the image raster are updated. The 

between the brush strokes applied by the user and the effect brush profiles are selected at the layer at which painting is 

of the painting on the underlying image raster. 5 performed, prior to the step of painting. Because there may 

^ . Jt ■ be many spatial transforms between the ViewElement layer 

Tims, in order for the changes that are made to an image and jj/j&e^t u me ofile of me brush al the 

raster during paintmg to be displayed correctly, an adjust- viewElemcnt k not match mat at mc EditElement 

ment is made to the brush strokes so that, after their i aycr . For example, a thin line drawn on an object that has 

application to the image material, the desired displayed been reduced in size by ten app ear thicker once the 

image may be obtained. Any spatial transforms that have 10 ob - ect hafi been relurned t0 its original size, 

been applied to the image raster between the level of the 0ne embodiment of a method that may be used to 

image raster material and the viewing level are applied in the transform the brush profile such that the appropriate pixels 

inverse to the brush strokes to ensure that the correct brush m modified in the image raster is shown in FIG. 7. At step 

strokes are applied to the image raster. Thus, if a series of yil, a CurrentElement variable is initialized to the selected 

spatial transforms have scaled the image raster by V4, and 15 EditElement. At step 174, a Brush matrix is initialized to the 

rotated it by 45 degrees to achieve the viewed composite identity matrix. At step 176, it is determined whether the 

image, the brush strokes would be applied spatial transforms CurrentElement and the ViewElement are the same. If not, 

to multiply them by 2 and rotate them by -45 degrees. at step \^ ^ brush matrix is multiplied by the current 

FIG. 6 illustrates one embodiment of a method for trans- elements spatial transformation matrix. The process pro- 

forming the brush strokes between the viewed image layer 20 ceeds to step 180, where it is determined whether there is a 

and the edited image raster. The brush strokes typically may Parent element associated with the CurrentElement. If so, at 

be defined as lines, having start and end coordinates, or as step 182 the CurrentElement is set to the Parent element, and 

discrete pixels having pixel coordinates. In the method, an the process returns to step 176. When it is determined at step 

EditElement is the element that is selected for editing, and 176 that the CurrentElement is equal to the ViewElement, or 

the ViewElement is the element that is currently being 25 when it is determined at step 180 that there is no Parent 

viewed. At step 152, a CurrentElement variable is set equal element, then at step 184 the inverse of the calculated brush 

to the EditElement. At step 156, the inverse spatial transform matrix is applied to the brush stamps to provide the modified 

of the CurrentElement is applied to the brush stroke. In brush profile. 

particular, the coordinates of the pixels that define the brush One example of a brush profile 300 is shown origin 

stroke are transformed according to the inverse spatial 30 centered in FIG. 8. Because the brush profile has a canonical 

transform to provide updated pixel coordinates. At step 158 ellipse shape, the geometry used to apply the transforms to 

it is determined whether the CurrentElement is the ViewEle- the brush strokes is slightly different than that described 

ment. If it is the view element, then all inverse spatial above in Equations I and II. The equation of a connonical 

transforms have been applied to the brush stroke, and the ellipse of a circle with radius of one is denoted by the 

process proceeds to step 164. If not, the process continues to 35 equation x 2 +y 2 «l . This equation may be rewritten as shown 

step 160 where it is determined whether there is a Parent below in Equation III: 

element for the CurrentElement. If there is a Parent element, Equation III: 
the process proceeds to step 162, where the Parent element 

is copied to the CurrentElement, and the process returns to [ l 0 1 [ jt l 

step 156. If there is no Parent element then the element was 40 ^^[oijUj 
the highest in the hierarchy, the Parent element value is null, 
and the process proceeds to step 164, where the process 

* Draw Brush stroke draws the pixel coordinates at the Applying the successive inverse transforms to the above 

transformed Brushstroke(x,y) locations. equation may be performed using below Equation IV, where 

When applying the spatial transforms and inverse spatial 45 s is a symmetric matrix: 

transforms to pixel coordinates, the general equations pro- Equation IV: 
vided below are used. For ease of explanation, assume the 

spatial transform is a 2x2 spatial transform A defined by t jc y>sf * 1 = 1 

Eauation I below: „ I > J 



Equation I below: 
Equation I: 



50 



The above equation may be used to apply successive 
transforms to the brush stamp 300. However, an accommo- 
dation is also made for the changes in the radius, angle and 
55 flatness of the brush stamp as a result of the transforms. The 
Applying the spatial transform to the (x,y) coordinates of variables defining the brush stamp are R, identifying the 
a pixel would yield updated pixel coordinates (ax+by, radius of the stamp, d, identifying the minor axis, i.e, 
cx+dy). If ad*bc, the inverse spatial transform of A is flatness, of the stamp, and * identifying the angle of the 
determined by the below Equation II: stamp. 

Equation II • 60 s * nce ttie mat " x S JS symmetric, it can be shown that there 

are two orthonormal vectors at and bf such SafoX^t, 
t r d -b] Sb=X 2 bf where X^X^O. 

A' 1 = ad _ bc \ c a \ Smce a t , b | are orthogonal, [xy]=ja \ +kb f for some j and 

a k, where j and k are any real numbers. The equation used to 

65 transform the brush stamp using the symmetric matrix S to 
While the above process may be used to ensure that the obtain updated R, d and § values (where § is the angle 
angle and extent of the brush strokes is applied appropriately between R and the x axis) is provided below in Equation V: 
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Equation V: 



13 



■ +k 2 = R 2 where: 



1 j IT 



d = R(l- F) 



A layer based paint system that operates using an image 
processing data structure, such as a tree, has been described. 
The image processing data structure is formed from linked 
image elements and process elements. By separating image 
elements from process elements, the image raster at an 
image element may be modified without un-doing any 
processing that is done at higher layers of the composite 
image. As a result, painting may be performed while view- 
ing the effects of subsequent processes on the painted image 
element. 

The image raster of an image element also may be directly 
modified using the paint process. Each image element may 
have one or more spatial transform performed on the image 
element prior to its display at a painting level. The coordi- 
nates of the painted brush strokes are transformed into 
coordinates of the image raster by iteratively tracing back 
through the image processing tree and applying the spatial 
transforms in the inverse to the brush strokes. The brush 
profile also may be adjusted in a similar mainer. 

Having now described a few embodiments of the 
invention, it should be apparent to those skilled in the art that 
the foregoing is merely illustrative and not limiting, having 
been presented by way of example only. Numerous modi- 



10 



20 



25 



14 



fications and other embodiments are within the scope of one 
of ordinary skill in the art and are contemplated as falling 
within the scope of the invention. 
What is claimed is: 

1. A method of displaying a composite image represented 
by a hierarchically ordered coupling of elements, the ele- 
ments including a reference to a parent element and refer- 
ences to any child elements, the elements apportioned into 
classes including image elements and process elements, the 
image elements including an image raster and at least one 
transform, the method comprising the steps of: 

a) designating all the elements at the lowest level of the 
hierarchically ordered coupling of elements as current 
working elements; 

b) for each current working element that is an image 
element, transforming the image raster using the asso- 
ciated image element transform; 

c) for each current working element that is a process 
element, processing the coupled children according to 
a process associated with the process element; " 

d) designating the parent elements of the working ele- 
ments as the working elements; 

e) repeating steps b through d until all of the elements in 
the image processing tree have been processed; and 

f) displaying an image defined by the image processing 
tree on the display screen. 
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